Systems and methods for navigation and visualization

ABSTRACT

A system for visualizing a surgical site is provided. The system includes a robotic mechanism for performing a procedure on a patient, an imaging device coupled to the robotic mechanism, the imaging device configured to provide image data of a site of interest, and a computing device coupled to the imaging device. The computing device includes one or more processors and at least one memory device configured to store executable instructions. The executable instructions, when executed by the processor, are configured to receive the image data of the site of interest, track motion patterns of the site of interest in the received image data, filter the received image data to remove line-of-sight restrictions therein and alter pixels therein based on the tracked motion patterns, and generate an output frame from the filtered image data. The system also includes a presentation interface device coupled to the computing device and configured to present the output frame for visualization of the site of interest.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/452,775, filed Oct. 29, 2021, which is a continuation of U.S. patentapplication Ser. No. 16/986,467, now U.S. Pat. No. 11,317,974, filedAug. 6, 2020, which is a continuation of U.S. patent application Ser.No. 16/113,666, now U.S. Pat. No. 10,765,484, filed Aug. 27, 2018, whichis a continuation of U.S. patent application Ser. No. 15/299,981, nowU.S. Pat. No. 10,058,393, filed Oct. 21, 2016, which claims the benefitof U.S. Provisional Application No. 62/369,821, filed Aug. 2, 2016, andU.S. Provisional Application 62/244,460, filed Oct. 21, 2015, each ofwhich is hereby incorporated by reference in their entirety.

BACKGROUND

The field of the disclosure relates generally to visualization andnavigation, and more specifically, to methods and systems forvisualizing sites that do not have direct line of sight to a user.

Generally, clear visualization is important when performing detailedtasks such as driving, operating machinery, or performing surgery. Forexample, surgical procedures require direct line of site to prepare andconduct the surgical procedure to ensure accuracy. To reduce thecomplications during the surgical procedure, surgeons attempt tominimize any disturbances to body. Those disturbances can includeminimal incisions that reduce the size surgical site, which in turn canlimit the field of view for the surgeon. Accordingly, a need exists forvisualization and navigation that provides feedback to a user (e.g., aphysician/surgeon) while performing tasks (e.g., preoperatively,intraoperatively, and postoperatively) to increase the accuracy andefficiency of the task.

BRIEF DESCRIPTION

In one aspect, a robotic system for navigation of a surgical site isprovided. The robotic system includes a computing device coupled to apresentation interface, a procedural component, and a communicationsinterface. The computing device is also coupled to a first imagingdevice configured to provide imaging data of a surgical site. Thecomputing device is also coupled to a second computing device that isconfigured to provide a second type of imaging data of the surgical sitethat is different that the imaging data of the first imaging device. Thecomputing device is configured to co-register the imaging data to createa surgical site image for display to a surgeon on the presentationinterface.

The features, functions, and advantages that have been discussed can beachieved independently in various embodiments or may be combined in yetother embodiments, further details of which can be seen with referenceto the following description and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a robotic system 100 used in surgicalprocedures.

FIG. 2 is a perspective of an exemplary procedural component that may beused with the system shown in FIG. 1 .

FIG. 3 is a perspective of an alternative procedural component that maybe used with the system shown in FIG. 1 .

FIG. 4 is a perspective of an alternative procedural component that maybe used with the system shown in FIG. 1 .

FIG. 5 is a perspective of a patient undergoing a procedure using thesystem shown in FIG. 1 .

FIGS. 6A and 6B are exemplary images produced by the system shown inFIG. 1 .

FIG. 7 is a perspective view of a portion of a spine having markers thatmay be sued with the system shown in FIG. 1 .

FIG. 8 is a cut-away perspective view of an exemplary temperatureimplant for use with the system shown in FIG. 1 .

FIG. 9A is an exemplary image produced by the system shown in FIG. 1 .

FIG. 9B is a schematic of the exemplary image shown in FIG. 9A.

FIGS. 10-12 are exemplary data flow diagrams of filtering imagesperformed by the system shown in FIG. 1 .

DETAILED DESCRIPTION

The systems and methods described herein enable accurate navigationduring surgical procedures. The systems and methods described hereinprovide landmark information inside the body of a patient during asurgical procedure. As used herein, the term “tissue” or “body tissue”refers to a group or layer of similarly specialized cells that togetherperform certain special functions and can refer to any type of tissue inthe body including, but not limed to, bone, organs, cartilage, muscles,skin, fat, nerves, and scars. As used herein the terms “procedure” or“surgery” refers to an operation performed on a patient to investigateand/or treat a pathological condition.

FIG. 1 is a block diagram of a robotic system 100 used in surgicalprocedures. System 100 includes a computing device 102, user inputcontrols 104, a presentation interface 106, a communications interface108, an imaging device 110, and a procedural component 112 having atleast one end effector 114. In some embodiments, system 100 iscommunicatively coupled (e.g., through an electrical wire or cable, orwirelessly through Bluetooth or Wi-Fi) to additional operating roomsystems including, but not limited to, monitoring surgical room controls120, surgical monitoring systems 122, and additional surgical systems124 as well as an operating table or bed 220. The robotic system 100could attach to the floor, be table mounted, and/or mounted to thepatient. As further described herein, system 100 may be attached to ormoving relative to the patient's body tissue. Moreover, system 100 couldbe a micro-robot that would be ingested or placed within the patient'sbody, including sensors that could communicate wirelessly and/orrecharge via electromagnetic radiofrequency motion, ultrasound,capacitive coupling, and the like. Non-limiting examples of roomcontrols 120 are HVAC (temperature, humidity), Oxygen, Nitrogen, CarbonDioxide, and lighting and non-limiting examples of surgical monitoringsystems 122 include cardiac, hemodynamic, respiratory, neurological,blood glucose, blood chemistry, organ function, childbirth, and bodytemperature monitoring. Additional surgical systems 124 include, but arenot limited to, anesthesia (e.g., oxygen, carbon dioxide, nitrogen,nitrous oxide, etc.), endoscopy, arthroscopic, electromagnetic guidance,oncology, navigation, arthroscopy, tissue ablation, ultrasound ablation,cardiac, stent, valve, cardiovascular, ESW, inhalation, urologic,cerebrospinal fluid, synovial fluid, OB/GYN, ENG, neurosurgery, plasticsurgery, pulmonary gastroenterology, and IV infusion systems. One havingordinary skill in the art will understand that system 100 may be used ina macroscopic manner and/or a microscopic manner, looking at cellularchemistry.

Computing device 102 includes at least one memory device 120 and one ormore processors 122 (e.g., in a multi-core configuration) that iscoupled to memory device 120 for executing instructions. In someembodiments, executable instructions are stored in memory device 120.Further, processor 122 may be implemented using one or moreheterogeneous processor systems in which a main processor is presentwith secondary processors on a single chip. As another illustrativeexample, processor 122 may be a symmetric multi-processor systemcontaining multiple processors of the same type. Processor 122 mayperform partial processing and receive partial processing by a processorand/or computing device communicatively coupled to computing device 102to enable cloud or remote processing. Further, processor 122 may beimplemented using any suitable programmable circuit including one ormore systems and microcontrollers, microprocessors, reduced instructionset circuits (RISC), application specific integrated circuits (ASIC),programmable logic circuits, field programmable gate arrays (FPGA), andany other circuit capable of executing the functions described herein.In the exemplary embodiment, processor receives imaging information fromdevice 110 and creates co-registered images for display on interface 106as well as providing movement limitations to component 112 based theimaging information.

In the exemplary embodiment, memory device 120 is one or more devicesthat enable information such as executable instructions and/or otherdata to be stored and retrieved. Memory device 120 may include one ormore computer readable media, such as, without limitation, dynamicrandom access memory (DRAM), static random access memory (SRAM), a solidstate disk, and/or a hard disk. Memory device 120 may be configured tostore, without limitation, application source code, application objectcode, source code portions of interest, object code portions ofinterest, configuration data, execution events and/or any other type ofdata. In some embodiments, memory device 120 retains or stores limitedor no information locally but stores information on a devicecommunicatively coupled to system 100 to enable cloud storage.

In some embodiments, computing device 102 includes a presentationinterface 106 that is coupled to processor 122. Presentation interface106 presents information, such patient information and/or images (e.g.scans), to a user/surgeon. For example, presentation interface 106 mayinclude a display adapter (not shown) that may be coupled to a displaydevice, such as a cathode ray tube (CRT), a liquid crystal display(LCD), an organic LED (OLED) display, and/or an “electronic ink”display. In some embodiments, presentation interface 106 includes one ormore display devices. In the exemplary embodiment, presentationinterface 106 displays surgical site data that is received from imagingdevice 110 and created by processor 122. The surgical site data may bedisplayed on presentation interface 106 and/or in any format thatenables user view to surgical site information including but not limitedto, glasses, a heads up display positioned within a surgical helmet, aretinal display that projects information onto the user's retina, and amonitor located within the operating room or some other remote location.In some embodiments, presentation interface 106 projects images fromsystem 100 directly into the retina of a surgeon. In some embodiments,surgical site data is provided to the surgeon with audible commands tohelp direct the surgeon during a procedure.

In the exemplary embodiment, computing device 102 includes a user inputinterface 104. In the exemplary embodiment, user input interface 104 iscoupled to processor 122 and receives input from a user/surgeon. Userinput interface 104 may include, for example, a keyboard, a pointingdevice, a mouse, a stylus, a touch sensitive panel (e.g., a touch pad ora touch screen), a gyroscope, an accelerometer, a position detector,and/or an audio user input interface. In some embodiments, user inputinterface 104 is a haptic feedback system that provides feedback (e.g.pressure, torque) from a procedural component 112. In some embodiments,a single component, such as a touch screen, may function as both adisplay device of presentation interface 106 and user input interface104. In one or more embodiments, user input interface is a sensor thatsenses vibration, heat, thermal properties, and the like.

In one embodiment, user input interface 104 is one or more sensorscoupled to a surgeon that are configured to detect muscle movement suchthat the procedural component 112 and/or end effector(s) 114 willrespond to the detected muscle movements. In some embodiments, sensorsare positioned on the skin of a surgeon or user so that the sensor candetect either mechanical (e.g., physical movement) or electrical signalsof the muscles and/or nerves. Such a system enables a surgeon to performa procedure remotely without the use of instrumentation directly coupledto the procedural component 112 and/or end effector 114. In someembodiments, a camera (i.e., imaging device 110) is utilized inconjunction with the sensors to determine and/or track surgeon movementpatterns to provide a more efficient determination of surgeon movements.

In the exemplary embodiment, computing device 102 includes or is coupledto a communication interface 108 coupled to processor 122. Communicationinterface 108 communicates with imaging device 110, procedural component112, and/or remote computing systems (not shown) such as mobile phonesand/or tablets. To communicate with imaging device 110, proceduralcomponent 112, and/or remote computing systems, communication interface108 may include, for example, a wired network adapter, a wirelessnetwork adapter (e.g. Bluetooth, Wi-Fi), and/or a mobiletelecommunications adapter. In the exemplary embodiment, communicationinterface 108 and presentation interface 106 enable remote conferencingof a procedure with system 100. For example, a surgeon can receiveguidance during a procedure from a remote surgeon, assistant, or medicalsales representative during a procedure. Additionally, system 100includes sensors of components 112 that sense movement of components andprovide feedback to system 100 enabling system 100 to provide signals toprovide tactile feedback and/or alarms to a surgeon through device inwhich the surgeon is interacting. Imaging device(s) 110 can provideremote users to visualize what is occurring in the procedure inreal-time, while allowing the surgeon to interactively communicate withthose remotely connected. Imaging device(s) 110 may also providepreoperative and/or postoperative images. Moreover, different types ofimaging devices (e.g., fiberoptic, light, acoustic, laser, etc.) may beutilized intraoperatively.

Additionally, the remote conferencing described above can also beutilized to enable remote inventory management. For example, a medicaldevice company or representative can utilize imaging device(s) 110 toview the inventory present in an operating room, or outside theoperating room in a secure location (e.g., pharmacy, stock room), todetermine what devices and/or objects have been utilized during aprocedure. In some embodiments, an imaging device 110 (e.g., camera)scans an inventory system (e.g., cart) to determine, via processor 122,which objects are no longer present and were utilized during aprocedure. It should be noted that system 100 can determine inventorylevels by utilizing additional sensors. For example, in someembodiments, system 100 is coupled to a scale that weighs the inventoryto determine missing items. In one embodiment, sensors are utilized toprovide feedback to determine missing inventory based on displacement,an empty space in a known inventory location, and/or a changed shape ofa stack or collection of inventory. Alternatively, system 100 isconfigured to track inventory using Automatic identification and datacapture (AIDC) sensors configured to provide device information byreceiving information from the inventory that includes, but it notlimited to including, bar codes, Radio Frequency Identification (RFID),biometrics, magnetic stripes, Optical Character Recognition (OCR), smartcards, and voice recognition.

The device utilization is processed by system 100 and transmitted, viacommunication interface 108, to a hospital billing department, medicaldevice supplier, practioner, clinic, and/or any other entity necessaryto track device usage (e.g., insurance, procedure payor, and governmentreporting entity). It should be noted that system 100 is configured totrack inventory systems within the medical and/or hospital including butnot limited to, implants, surgical instruments, disposable medicaldevices, pharmaceuticals, and bracing. Additionally, the inventorycontrol features described herein could be utilized by any systemneeding inventory control outside of the medical field.

In some embodiments, system 100 utilizes movement patterns fornavigation (e.g., surgical navigation, vehicle navigation, etc.). In asurgical context, the actual movement or stimulation of tissue thattwitches, moves, or goes in a motion pattern can be tracked by system100 and used to navigate (i.e., understand where soft tissue is locatedrelative to soft tissue and/or bone). For example, if electricalstimulation is used to stimulate a muscle or nerve to twitch, system 100can track these movement patterns to determine where the nerve or muscleis located. In another exemplary embodiment, a spasm in the blood vesselcan be created to determine where the blood vessel is located to createpatterns of navigation. System 100 can also be used at a microscopiclevel to create navigation at a cellular level where cell membranesand/or cell receptors are stimulated. As movement patterns are tracked,system 100, using methods described herein, could remove pixels and/orenhance or change a visualization. For example, if tissue, cells, and/ormembranes are stimulated to move, system 100 could eliminate or removethose pixels

In some embodiments, aspects of system 100 utilize extensor opticalsystems. Numerous optical sensors are known to those of ordinary skillin the art. For example, if there are opacities in the way of theoptical sensor such as cloudiness, bleeding, or synovial fluid, anoptical sensor may inaccurately note properties, such as pH or pressure.By removing opacities, system 100 improves the functioning of theoptical sensors. In some embodiments, system 100 changes light to colorthe frequency or intensity by strobing, flashing on/off, and/ordisplaying different intensities as it reflects (i.e., albedo). System100 may also remove tissues that reflect differently based on lightcolor, frequency, intensity, and/or strobe.

FIGS. 2-4 are schematic diagrams of exemplary procedural components 210,230, and 250 that may be used with system 100, shown in FIG. 1 . FIG. 2is a schematic diagram of an exemplary procedural component 112 in theform of a telemanipulator 210. Telemanipulator 210 receives operationalinstructions from a user/surgeon operating input interface 104. In suchan embodiment, the user/surgeon is capable of performing normal surgicalmovements while arms 212 carry out those movements using end-effectorsand manipulators 214 to perform the actual surgery on the patient 200.It should be noted that utilizing a telemanipulator 210, the surgeondoes not have to be present in the operating room, but can be anywherein the world, leading to the possibility for remote surgery. In someembodiments, telemanipulator 210 includes an imaging device 110 (e.g.endoscope) within an arm 214.

FIG. 3 is a schematic diagram of an alternative procedural component 112in the form of a user movable robotic arm 220. In the exemplaryembodiment, a surgeon manipulates arm 220 by moving an end effector 222into place by movement of handle portion 224. During a procedure, thesurgeon utilizes arm 220 and more specifically end effector 222 withlimitations that are imposed by computing device 102 and/or processor122 with information obtained from imaging device 110.

FIG. 4 is a schematic diagram of an alternative procedural component 112in the form of a manual surgical tool 230. In such an embodiment, tool230 is manufactured to be portable (i.e. hand-held) such that thesurgeon can manipulate the tool 230 during a procedure. Tool 230includes an end effector 232 capable of performing surgical functions ona patient. In an embodiment, tool 230 can be at least partiallysterilized. In another embodiment, tool 230 includes surgical drainsthat cover part of tool 230 and sterilize parts in an operating room,such as sleeves, drapes, and the like.

It should be noted that system 100 is configured to complete an entiresurgical procedure utilizing only system 100, inclusive of theprocedural component 112 which non-limiting examples are represented bytelemanipulator 210, arm 220, and tool 230. As noted above, proceduralcomponent may include one or more end effectors 214, 222, and 232 toperform to actions needed to perform the surgical procedure. The actionsof component 112 can be any surgical action including, but not limitedto, sewing, stitching, stapling, cutting, sawing, cauterizing, grasping,pinching, holding, tensioning, moving, implanting, removing, viewing,sensing force, sensing pressure, and tying. The end effectors 214, 222,and 232 can be any end effector needed for performing the surgicalactions including, but not limited to, forceps, needles, needle drivers,retractors, clip appliers, probe graspers, cardiac stabilizers,balloons, tissue dissectors, saws, knives, mills, reamers, coagulationdevices, lasers, ultrasonic transducers/probes, cautery instruments,scalpels, staplers, scissors, graspers, and sealers.

In the exemplary embodiment, system 100 includes at least one imagingdevice 110. As shown in FIG. 5 , imaging device 110 can substantiallysurround or be positioned adjacent patient 200 undergoing a procedure(i.e., intraoperative) performed at least in part by proceduralcomponent 112 and/or end effector 114. In the exemplary embodiment,imaging device 110 includes a C-arm 300 coupled to a scanner 302. In oneembodiment, scanner 302 is a gamma camera configured to detectradioactive tracers inside the body of patient 200. Alternatively,scanner 302 is any device that is capable of scanning and/or detectingenvironmental information of patient 200, a surgical site, and/or andoperating room including, but not limited to, endoscopy, fluoroscopy(e.g. X-ray, CT, CAT scan), laser or ultrasonic Doppler velocimetry,projection radiography, MRI, SPECT, PET, ultrasound, infrared,elastography, tactile imaging, photoacoustic imaging, thermography,tomography, echocardiography, NIRS, and fNIRS. In an embodiment,environmental information scanned and/or detected via a plurality oftechniques may be combined (e.g., combining visible light wavelengthsand infrared for imaging technologies). In some embodiments, imagingdevice 110 may have one or more scanners 302 located at, near, adjacent,or in a surgical site to perform imaging. In one embodiment, imagingdevice 110 includes a scanner 302 that is configured to locateprocedural component markers 250 positioned on procedural components112. Marker 250 information is transmitted to computing device 102 todetermine component location relative to patient 200, other componentsof system 100, and/or the operating room. In the exemplary embodiment,system 100 includes an imaging device 110 that provides holistic imagingof a surgical site to provide users with the necessary imaging tocomplete a procedure. For example, in a knee arthroplasty procedure, theimagine device(s) 110 used in the procedure would provide the surgeonimages of the knee as well as relative joints (e.g., hip and ankle) toensure an effective procedure. In such a procedure, system 100 wouldproduce a three dimensional model of the hip, knee, and ankle to provideviews of the procedure in all angles including anterior-posterior (AP)views and medial-lateral (ML) views. While a non-limiting example of anarthroplasty procedure is provided, it should be noted that the holisticimaging could be utilized with any type of procedure utilizing system100.

In preparation for use of the robotic system 100, a calibration isrequired to ensure that accuracy of the end effectors 114. Typically,imaging (e.g. CT) of the patient is done preoperatively and the imagesare loaded into system 100. In some instances, during the calibration ofsystem 100, while patient 200 is in the operating room and on theoperating table, patient markers are placed or pinned into the body toprovide landmark information. The system 100 is configured to associatethe patient landmark information with the images provided preoperativelyto provide a map of a surgical site as shown in FIG. 9 .

In some embodiments, calibration is aided by coupling the robotic systemto the surgical table and/or the patient. Coupling system 100 to asurgical table or bed provides system 100 relative patient positioninformation as the patient is moved during a procedure. Referring toFIG. 2 , procedural component 112 can be configured to couple directlyto table 220 at a base 230 of telemanipulator 210 or at or near arm(s)212 of telemanipulator 210 via coupling mechanism 232. Referring to FIG.3 , movable robotic arm 220 can be configured to have one or morerecesses or docks 240 provided within arm 220 to couple directly intotable 220. In addition to being coupled to table 220. Components 112 canbe coupled directly to the patient. In some embodiments, custom moldedor 3-D printed devices 250 can be created for each patient to be wornduring a procedure. To create such devices 250, preoperative scans aretaken of a surgical site and custom fit devices 250 would bemanufactured to be placed on the patient. In one such non-limitingexample, as is shown in FIG. 3 , for a brain surgery, a helmet 250 iscustom manufactured for patient 200 and includes attachment portions 252that provide a coupling point for components 112 as well as apertures254 for performing the procedure. Devices 250 can be manufactured to fitany body portion undergoing a procedure including, but not limited to,an abdomen, leg, knee, foot, ankle, neck, back, torso, arm, hand, head,and face.

It should be noted that procedural components can be configured to dockor electrically couple into surgical tables or beds such thatcommunication from each can be transmitted back and forth via acommunications interface. In some embodiments, procedural components 112rigidly couple to the table or bed while other embodiments provide anelectrical wire, cable, or wireless (e.g., Bluetooth or Wi-Fi) couplingbetween the components 112 and the table. In addition, in someembodiments, procedural components 112 rigidly couple directly to apatient or to surgical instrumentations utilized during a procedure(e.g., surgical robot, instrumentation, visualization system, retractor,electrosurgery system, knife, saw, and mill). In some embodiments,procedural components 112 are used in a transmitter/physician office orother locations such as insurance companies and the like.

In the exemplary embodiment, a radioactive tracer is inserted into thebody and landmark information of the patient is determined by aradioactive detector (e.g. scanner 302) of system 100 and provided to asurgeon via presentation interface 106 to increase the efficiency andaccuracy of the procedure. As is commonly known, radioactive tracersemit gamma rays from within the body. These tracers are generallyshort-lived isotopes linked to chemical compounds (e.g.radiopharmaceutical) that enable examination of specific physiologicalprocesses and/or anatomical landmarks. In some embodiments, the tracersare given through an intravenous injection (e.g. IV), inhalation, ororally. In some embodiments, the tracers are optical and/orbiodegradable.

In the exemplary embodiment, Technecium-99m is used as the radioactivetracer. Technetium-99m emits 140 key gamma rays with a half-life ofapproximately 6 hours that exits in the form of pertechnetiate ion(TcO4). Alternatively, any radioactive tracer could be used with thesystems and methods described herein, including but not limited to,Bismuth-213, Calcium-47, Carbon-11, Cesium-137, Chromium-51, Cobalt-57,Cobalt-60, Copper-67, Dysprosium-165, Erbium-169, Fluorine-18,Gallium-67, Holmium-166, Indium-111, Iodine-123, Iodine-125, Iodine-131,Iridium-192, Iron-59, Irridium-192, Krypton-81m, Lutetium-177,Molybdenum-99, Nitrogen-13, Oxygen-15, Palladium-103, Phosphorus-32&33,Potassium-42, Rhenium-186, Rhenium-188, Rubidium-82, Samarium-153,Selenium-75, Sodium-24, Strantium-85, Strontium-89, Strontium-92,Sulfur-35, Technecium-99m, Thallium-201, Uranium-235, Xenon-133,Ytterbium-169, and Yttrium-90.

In some embodiments, the tracers are detected by a gamma camera, whichrecognize photons enabling a view of internal landmarks of a patientfrom many different angles. In such an embodiment, the camera builds upan image from the points from which radiation is emitted and the imageis enhanced by system 100 and viewed by a physician on monitor 106. Inan alternative embodiment, a Positron Emission Tomography (PET) isperformed in which a PET camera detects emission of two identifiablegamma rays in opposite directions to identify landmarks. In yet anotherembodiment, myocardial perfusion imaging (MPI) is performed to identifylandmarks. In some embodiments, images having landmarks identified withtracers (e.g. gamma, PET, MPI) are utilized with a computerizedtomography (CT) scan and the images are co-registered (e.g. layered) bysystem 100 to provide complete landmark information. It should be notedthat the tracer images can be co-registered with any other type ofimaging (e.g. ultrasound, x-ray, and MRI) to produce landmarkinformation. In an embodiment, the tracers are used for ultrasoundnavigation with or without radiation. The ultrasound navigation may beutilized for surface mapping or registering anatomical data points.

During robot assisted surgery, such as surgery utilizing system 100, itis necessary to have multiple fixed points for recognition by trackersto allow for navigational computation. Currently, in the case ofarthroplasty, invasive pins are inserted into the bones to calibrate therobotic system and to create landmarks. In the exemplary embodiment, thelandmark information found via tracers is utilized for calibration andnavigational computation. In such an embodiment, after theradiopharmaceutical (e.g. technetium) is introduced into the system, thetracer is taken up by osteoblasts and noted on resulting images. Often,the tracer appears dark on an image and can be known as a hot spot.These locations are co-registered with other images, by system 100 andthe system 100 performs calculations using known algorithms to determinekey distances and/or boundary layers for surgery.

FIGS. 6A and 6B are illustrations of an exemplary image 400 created,displayed, and/or utilized by system 100. In the exemplary image 400, abone scan created with the use of a gamma camera is shown with multiplehot spots 402, indicative of the location of the radiopharmaceutical. Insome embodiments, a (e.g. surgeon, physician's assistant, nurse) createsa cut within tissue (e.g. bone) that would promote osteoblast formationand hot spot formation as the osteoblast will absorb the tracer. In suchan embodiment, the tracer can be injected and/or placed directly on thecut such that the tracer remains substantially in place and avoidssystemic introduction of the tracer. In an embodiment, different tracersmay be absorbed into different tissues such as thyroid, liver, kidney,and the like. The system 100 is also capable of creating, displaying,and/or utilizing three-dimensional images using ultrasound, radializedisometry, and/or three-dimensional views anterior, posterior, andcircumferential.

In one embodiment, as shown in FIG. 7 , in addition to, and/or insubstitution of, creating tissue cuts, a user places one or more tissuemarkers 420 in discrete locations within the body. The tissue marker 420can be injected, filled, or formed with a tracer to provide locationinformation (e.g., via a thermogram). In one embodiment, the marker 420is a scaffold that is formed to adhere to the contours of body tissue.The scaffold 420 may be formed to adhere directly to the tissue and/orbe affixed with a fixation substance (e.g. surgical adhesive,bioadhesive, glue, fasteners, ultrasonic welding, other tissuereconstruction repair approaches). Alternatively, tracers can becompounded with an agent (e.g. PLLA, collagen) and positioned on thetissue with or without the use of a fixation substance. It should benoted that the markers can be biodegradable such that the tissue markercan remain in the body after the surgical procedure. Additionally, themarkers can be fabricated from a non-biodegradable material. In someembodiments, the markers include a wireless transmitter (e.g. RFID tag,Bluetooth) that provides, at minimum, location information for inventoryand/or complication risk assessment. Additionally, the markers can besensors in the tissue.

In one embodiment, sensors 422 are positioned within a procedure site.The sensors 422 are configured to detect and transmit non line of sightsurgical data to the surgeon, through system 100. Although the sensorsare configured to communicate with system 100 in a wireless fashion, thesensors 422 can electrically couple to system 100 to communicatedirectly over a transmission line (e.g. fiber or metallic cable). In oneembodiment, the sensors 422 would act as Geiger counters and detecttracers within a particular location. In an embodiment, sensors 422 arepowered by capacitors. In another embodiment, sensors 422 are powered bybody flow across cell membranes (i.e., turning tissue into a battery) byutilizing electrical energy through the membranes through thermalapplication. It should be noted that the body flow could be accomplishedthrough synthetic tissue. In an embodiment, sensors 422 measuremicrocellular electrical gradients inside the cell. In some embodiments,sensors 422 are configured to detect force to provide feedback as to theforces exerted on and around tissue. In some embodiments, sensors 422measure electrical pulses emitted by the body (e.g. nerves) duringsurgery. In such embodiments, sensors 422 can detect pulses by EEG, EMG,micro electrode arrays, or other electrophysiological recording methods.In one embodiment, sensors 422 are provided such that somatosensoryevoked potentials are detected during a procedure. It should be notedthat sensors 422 can be any sensor that monitors environmental factorsincluding, but not limited to, force, acoustic, vibratory, density,pressure, optical, chemical, and electric. In some embodiments, sensors422 are optical and/or biodegradable. Sensors 422 may also be positionedon the skin of a patient or implantable in the body of the patient. Insome embodiments, sensors 422 partially degrade. For example, sensors422 could include a biodegradable coating or slowly degrade over aspecific time when exposed to water. The degradation may be hastened byheat, pH, and the like. In some embodiments, sensors 422 arerechargeable, such as through electromagnetic, optical, laser,ultrasound, vibratory, and/or thermal techniques.

In one embodiment, sensors 422 are configured to measure distances ofparticular tools. For example, as shown in FIG. 7 , an end effector 114(e.g. ultrasonic scalpel) may have an emitter 424 positioned at a fixedlocation. As end effector 114 is inserted in the body, system 100 canmonitor the distance of the tool relative to sensor 422. In theexemplary embodiment, the sensor is placed on at least a portion of thespinal cord 430. As the tool would progress internally towards thespinal cord 430, system 100 and/or processor 122 would lock-out(prevent) end effector from functioning and/or progressing to preventdisruption of spinal cord 430. In some embodiments, sensors detect toolsdirectly without the need for an emitter. For example, sensor may detectthe vibratory or acoustic energy emitted by tool to determine a locationof the tool relative to the sensor.

As noted above, system 100 can be calibrated such that the markers 420and/or sensors 422 provide boundaries such that disturbances to portionsof the body not part of the surgical procedure are minimized.Additionally, the markers 420 and/or sensors 422 can provide a secondarylayer of protection should a fault occur. In such an embodiment, themarkers and/or sensors can require the software of system 100 to performa first check of calibration and confirm navigation throughout theprocedure.

It should be noted that while methods and systems shown herein aredepicted for arthroplasty, the methods and systems described herein canbe utilized in any part of the body for any surgical procedure. Forexample, a tracer could be absorbed into organ (e.g. heart, gallbladder,etc.) which enables system 100 to create a boundary for proceduralcomponent 112 and/or end effector 114, such that the end effector doesnot work outside of the boundaries created using information obtainedfrom markers 420, sensors 422, and/or scanners 302. Accordingly, as thesoft tissue moves or is moved, system 100 can maintain an accuratelocation of the surgical site. Moreover, the methods and systemsdescribed herein can be utilized for robotics and/or haptics guided withpreoperative imaging via CT, MRI, ultrasound techniques, and the like.The method and systems described herein may also be utilized standaloneintraoperatively.

In the exemplary embodiment, system 100 receives instructions fromsoftware that enables processor 122 to determine boundaries within asurgical site and prevent (e.g., lock-out or disable) components 112and/or end effectors 114 from operating outside of the determinedboundaries. To this, if components 112 and/or end effectors 114 areprevented from continued operation due to a boundary, system 100 and/orprocessor 122 is configured to determine whether to move the component112 and/or end effector 114 or a portion (or all) of table 220 to enablefurther operation of the component 112 and/or end effector 114. In oneembodiment, the determination of what object to move is provided to asurgeon to enable manual intervention.

Alternatively, system 100 provides signals to the appropriate object(e.g., effector 114 or table 220) to enable the procedure to continue.The signals provided to the table 220 can be any signals that effect atable to re-position a patient as needed, including but not limited to,manipulating, rotating, torqueing, twisting, distracting, flexing,extending, elevating, descending, inflating or utilizing a retractor(i.e., internal or external) or bolster that aid in bringing objectsinto/out of a surgical site. Such a repositioning of objects enables asurgeon to optimize portions of a body relative to a portal or incision.For example, system 100 is configured to provide instructions tomanipulate vessel blood flow, organ position, or the position of anyother body part. The signals provided to the table 220 can also be anysignals that affect a table to move relative to a body part. The bodypart could also move relative to the table 220 and/or relative to othersystems (e.g., 120, 122, 124, and 220) coupled to system 100. Moreover,the table and the body part could both move together synchronously orindependently. In addition to providing instructions to manipulate table220, system 100 can also provide instructions to other systems (e.g.,120, 122, 124, and 220) coupled to system 100. For example, system 100manipulates a change in table 220, system 100 would also transmit asignal to adjust lighting or visualization to re-position to the newlocation of the surgical site. The signals described herein may enablecontrol and/or activation of aspects of system 100 through voicecommands, remote commands, imaging controls, and/or robotic activation.Aspects of system 100 may also be controlled with a robotic mechanismwith or without navigation or visualization.

Additionally, if processor 122 determined that a repertory change of thepatient would aid in the effectiveness of the procedure, a signal can begenerated and transmitted to the anesthesia system and/or anesthetist toalter the anesthetic (e.g., amount or type) given. In an embodiment,processor 122 generates and transmits a signal to dictate anesthesia to,for example, increase or decrease inflation of the lungs, change theblood pressure, heart beat rate, lung volume, and the like. Moreover,processor 122 is capable of controlling electrical currents fortranscutaneous electrical nerve stimulation (TENS) to stimulate muscularactivity and/or vascular activity and control position. For example,electrothermal vibrations could be used to stimulate tissue and/orelectrical soma.

FIG. 8 is a cut-away perspective view of an exemplary temperatureimplant 500 for use with system 100 shown in FIG. 1 . Implant 500includes a heating portion 502 that provides heating to tissue in directcontact with implant 500. Implant 500 also includes a cooling portion504 that surrounds and is adjacent heating portion 502 that preventsheating of tissue that is not in direct contact with heating portion502. Heating portion includes a surface 506 that is configured toincrease temperature as a result of a heating element 510 positionedwithin implant 500. Heating element 510 can be any element that producesheat to surface 506 including, but not limited to, metal, ceramic, andcomposite. In one embodiment, heat is produced from the use of vibratory(e.g. ultrasonic) energy. In the exemplary embodiment, surface 506 isfabricated to include a polymer but can include any material thatenables heat transfer including, but not limited to, metals and polymercomposites. In some embodiments, heating portion 502 includes a magneticsurface. Heating element 510 can also be any element that produceselectrical charges to an implant surface, cell membrane, tumor, orinfection, for example through microscopic membranes, cell membranes,nerve fibers, mitochondria, and/or intracellular.

To provide cooling to implant 500, cooling portion 504 includes a heatexchanger to reduce heating of tissue that is not in direct contact withsurface 506. In one embodiment, implant 500 is fabricated to be modular.In such an embodiment, implant 500 is fabricated to have multipleremovable sections such as a base section 520 and first modular section522. Modular section(s) 522 can be added or removed to increase orreduce the size of the implant 500 thus providing for a variable sizedimplant.

In the exemplary embodiment, heating element 510 and/or cooling portion504 is powered by a controller 530 and power source 532. In oneembodiment, power source 532 is a battery. Alternatively, power source532 is a power converter that converts power (e.g. A/C or D/C current)from a power source (e.g. outlet) into electrical signals for use by theheating element 510 and/or heat exchanger 504. Implant 500 also includessensors 534 configured to monitor environmental factors within theoperating site. In one embodiment, a sensor 534 is temperature sensorconfigured to monitor the temperature of implant 500 and/or the tissuein contact with surface 506 and/or the tissue adjacent to implant 500.Additionally, the sensors 534 can be any sensor that monitorsenvironmental factors including, but not limited to, force, acoustic,vibratory, density, pressure, optical, chemical, and electric. In someembodiments, implant 500 includes a timer (not shown) coupled tocontroller 530 that enables controller to selectively provide power toheating element 510 at predetermined times or intervals. In anembodiment, sensors 534 are internal sensors.

Sensors 534 are coupled to a communication interface 540 coupled toprocessor 542 and/or controller 530. Communication interface 540communicates with system 100, shown in FIG. 1 . In addition to providingoperating site information, communication interface 540 also receivesinstructions, remotely, for powering and adjusting parameters (e.g.temperature) of implant 500. To communicate with system 100,communication interface 540 may include, for example, a wired networkadapter, a wireless network adapter, and/or a mobile telecommunicationsadapter. Moreover, communication interface 540 may communicate via nervetransport, cellular membrane transport, and/or cellular signals.

In use, implant 500 is placed on tissue. In some embodiments, implant500 heats tissue for a predetermined amount of time and then removed.After removal, a thermogram scan can be taken to provide landmarkinformation to system 100. In one embodiment, implant 500 remainspositioned within the body to allow for selective heating of tissuethroughout the procedure. The resulting thermogram images can beco-registered with other images (e.g. CT, MRI, X-Ray, Gamma) taken ofthe patient to provide landmark information to system 100 and/or thesurgeon. In some embodiments, the thermogram can be utilized as analternative to the radioisotopes described above. The advantages of theuse of implant 500 and the resulting thermogram images is that landmarkinformation can be provided and registered without having direct line ofsite enabling the implant 500 to be positioned on the side, back, orinto the padding behind the arthroplasty as one would not require directline of site, which would impede the surgical procedure. In someembodiments, implant 500 uses electrical, thermal, and/or magnetictechniques to stimulate muscle, vessels, nerves, and the like tocontract over movement patterns. Landmark information can be detected bycreating boundary levels as well as navigation. For example, astimulator would move to detect where the movement is and then createboundary layers or guidance direction to a specific site.

In one embodiment, system 100 includes a scanner 302 in the form of avessel determination device. The scanner is configured to locate vesselsand flow rates in the body. The scanner 302 can include any flowdetermination technology for locating vessels including, but not limitedto ultrasonic flow meters and/or laser Doppler velocimeters. Once thevessels are located, positional information can be transmitted to system100 to be input into images and navigation calibration. In oneembodiment, system 100 determines the type of vessel (e.g. artery, vein,capillary) based on the size and/or flow rates of the located vessels.Vessels that are within a procedure site can be used as boundaries suchthat procedural component 112 and/or end effector 114 will be disabledwhen approaching the vessel to maintain patient safety during theprocedure.

FIG. 9A is an exemplary image 600 created by processor 122, shown inFIG. 1 , using information received from scanners 302 for display oninterface 106. FIG. 9B is a schematic representation of image 600. Image600 is created by co-registering multiple images created from scanners302. As can be seen in the image, a knee joint having a femur 604 andtibia 606 are created from an MRI. Image 600 also displays a vessel 602positioned behind the femur 604 and tibia 606 from flow determinationtechnology. Radioactive tracer information is shown by hot spots 608that were derived from a gamma camera or PET, ands sensors 422 ormarkers 610 can be located as well. Image 600 also includes thermographyinformation 612 from the patient. In one embodiment, image 600 includescutting guides 614 that display portions of tissue that will be removedby the surgeon. It should be noted that any and all of the landmarks,sensors, or makers that can be identified by system 100 can serve asboundaries or limitations on procedural components 112 and/or endeffectors 114. Additionally, any of the imaging and/or landmarkdetermination techniques described herein can be combined such thatprocessor 122 can co-register the information to produce images andinstructions for system 100.

In addition to image 600, processor and/or system 100 can also generatea composite image of the surgical site in the form of an animation or3-D rendering based on the information shown in FIG. 9 . Such an imagewould provide a reconstruction of the surgical site enabling the surgeonto rotate through the surgical site and visualize the entire site fromany desired angle. As noted above, the images produced would also enablea surgeon to receive a holistic view of a site. For example, while aportion of image 600 displays a portion of the knee, a surgeons' viewwould provide views of the pelvis, hip, foot, and ankle for relativeinformation. Such information is vital to determine how objects in thesurgical site (e.g., knee movement) affect remote portions of the body(e.g., hip, spine, and ankle). In addition to optical changes, theimages produced may also enable detection of electrical and/or motionpatterns.

In the exemplary embodiment, the images produced by system 100 alsoprovide a surgeon the ability to locate anatomical structures (e.g.,bones, organs, arteries, vessels, cartilage) in a surgical site anddenote those structures by color coding or labeling. Such structures canalso be removed or added to a view, via input interface 104, during aprocedure based on the surgeons' needs. Images produced by system 100may also be utilized for external ablation systems that utilizeultrasound, thermal, and like techniques to refine exact tissue locationfor ablation of tumors, treatment of infection with antibiotics toenhance growth, neurologic tissue, rewire neurons, and/or repair complexneurological bundles.

To decrease the trauma (e.g. pain, swelling, etc.) received or resultingfrom a surgical procedure, surgical markers and/or sensors can bepositioned at points of interest for a surgeon. The surgical markers aresubstances including a dye that fluoresce when exposed to ultravioletlight (UV). As an alternative to directly placing the surgical markerson tissue, tissue closure devices (e.g. suture, staples) can beimpregnated with the dye such that it is absorbed or transferred to thetissue that is in direct contact with the closure device. For example,in a revision arthroplasty procedure in which an infected jointreplacement component is being extracted and replaced, the surgeon canposition surgical markers on the incision or open tissue after he/sheextracts the joint replacement component and before closing the surgicalsite to allow the body to eliminate/fight the present infection. Whenpatient returns for the secondary procedure (e.g., placing a drug localinto the body), ultraviolet light can be used to locate former incisionlocations. Using the UV indicated locations, a surgeon can utilize theformer incision to open the surgical site. Utilizing a former incisioncan greatly reduce pain, inflammation, and trauma to the patient as scartissue generally forms at locations of former incisions, which has beenfound to be less traumatic to a patient than disturbances (i.e. cuts) tomuscle tissue.

In addition to the images created in system 100 from imaging devices110, system 100 can include software filter for filtering out materialand/or objects from an image that can restrict line of sight of a user(e.g., physician/surgeon, driver, machine operator, etc.). For example,the filters described herein would enhance the surgeon's ability tovisualize a surgical site during a procedure (e.g., arthroscopy) byfiltering opaque bleeds or blood flow and allowing the surgeon todetermine the location or source of the bleed. In another exemplaryembodiment, the filters described herein would enhance the driver'sability to visualize upcoming stretches of roadway by filtering fog,rain, and the like and allowing the driver to determine if hazards existon the upcoming stretches of roadway. When video is digitized, eachframe is represented by a two dimensional array. Each location in thearray represents a pixel and each pixel contains a set of valuesrepresenting color and other attributes. The filters described hereinmanipulate these pixel values. For example, the filters described hereinmay change pixels and/or remove pixels with or without electricalcharges or motion changes. It should be noted that these filters couldbe applied to any video format and coding methods including, but notlimited to, PAL, NTSC, MP4, and AVI.

In one embodiment, a first filter is utilized within system 100 and thefirst filter is configured to determine when blood exists in the salinesolution of a surgical site during surgery. This is accomplished bymonitoring a sequence of frames for a range of target colors that movein a particular pattern. In one embodiment, the first filter lowers thecolor values of blood to match the surrounding colors. Alternatively,the first filter lowers the intensity of color of blood (e.g. red) toenable the surgeon to better visualize the intended area. In someembodiments, the first filter lowers the color values and well aslowering the intensity of color. For example, one could see thereflective coefficient (e.g., albedo), albedo with different lightsources, and/or different movement creating the changes. In someembodiments, the first filter accomplishes the determination withvibratory changes, acoustic changes, moving cells that change, and/ormove or have specific electrical charges. The first filter could removethese pixels in the tissue/bone. In some embodiments, the target (e.g.,tissue) may be magnetized. In some embodiments, the filter bounces,changes, and/or reflects light. For example, the filter could enable auser to see around corners with reflective light by using an albedo orreflective coefficient.

In another embodiment, a second filter is utilized by system 100 toprovide images to a user. The second filter is configured to adjustparticular colors in an image by removing pixels in each frame that meeta predetermined criteria (e.g., blood flow). The pixels in the bufferwhich would be displayed on a standard image without the use of secondfilter would then be used in place of the obscured pixels. This wouldgive the second filter hysteresis which would allow it to use previousinformation to help recreate what is behind an object (e.g., blood,synovium, tissue fats, debris, bone fragments) that could obscure theview other objects of interest (e.g., soft tissue, cartilage, muscle,bone). It should be noted that the first and second filters could beused in combination to provide an accurate rendering of a surgical siteenabling a surgeon to selectively eliminate unnecessary objects fromtheir view. For example, multiple filters could be used to change orangepixels to red pixels. In some embodiments, one filter is a softwarefilter, for example a mechanical filter (e.g., film, prism, Fresnellens, UV lens).

In some embodiments, the second filter is utilized by generating abaseline image and comparing the baseline image to new images taken atpredetermined time increments (e.g., 0.1, 0.5, 1, 2, 30 seconds orminutes). In such embodiments, pixels could be compared to determineflow patterns of fluid or other moving objects. Additionally, system 100is configured to enhance objects in an image by receiving imaginginformation, comparing to known imaging information (e.g., pre-operativescan), determining the differential and adding necessary data to thereceived imaging information.

FIG. 9 is an exemplary flowchart 700 of a method of visualization foruse with the system 100 shown in FIG. 1 . In the exemplary embodiment,system 100 receives an image of a site. In some embodiments, an image isreceived by computing device 102 from an imaging device 110 and or inputinterface 104. In some embodiments, images are received by computingdevice 102 from a remote location through communications interface 108.

FIGS. 10-13 are images utilized with the method shown in FIG. 9 .

In some embodiments, system 100 utilizes filters to determine the sourceof the blood flow. Once system 100 and/or processor 122 determines thelocation or source of a blood flow, system 100 can indicate, on theimage, the source of the blood. Additionally, system 100 can alsoindicate a particular concentration of blood in the image at certainareas where the blood has stronger concentration. In some embodiments,system 100 can determine the velocity of the blood flow in one or morelocations, which may indicate the source of blood flow. Any of thedeterminations described above, can be indicated on an image, by system100 and/or processor 122, with indicia having non-limiting examples of acircle, marker, or color differentiation so that the surgeon can easilylocate the area of blood flow to allow for electrocautery, to locate thesource of the bleed, and/or to apply pressure for coagulation.

In one embodiment, a diagnostic ultrasound or B-Mode ultrasound head isimbedded into an imaging device 110 (e.g., camera) to enable overlayingthe information from the diagnostic ultrasound with the video image inreal time. This provides the surgeon a 3-dimensional view of thesurgical site as well as the direct camera view. Such a combination isuseful in removing blood or other debris from the surgeon's view withfilters. In some embodiments, a second portal or external ultrasound isutilized in conjunction with the camera to enable these filters as well.If either internal or external ultrasound is used, it is possible to useDoppler information to better detect blood for filtering. The filterswould work as previously mentioned for either removal the color or usinginformation from previous frames. The Doppler information is useful in amore precise determination of the location of the bleed. The filtershould also monitor for dramatic increase in the blood flow. If system100 determines an increase in blood flow has occurred, an alert istransmitted to the users that a bleed is occurring and being filteredout that might require attention. In one embodiment, the alert isprovided as an overlay warning the surgeon of the increase in bloodflow. Alternatively, system 100 can be configured to turn off filterswhen an amount of blood in the field of view exceeds a predeterminedthreshold. In some embodiments, system 100 is configured to filter outobjects that produce a reflection at or above a predetermined threshold.

In one embodiment, system 100 creates a charged particle filter used forfiltering out particular objects. In such an embodiment, system 100projects or transmits a number of charged particles (e.g., ions in agas) at a site that attach to a particular object. The charge isselected by determining an object to be filtered out and chargingparticles to a predetermined setting that would enable the particles toattach or couple to the object. Once the particles attach to the object,system 100 detects the location of the objects and filters out theinformation from the images generated.

In some embodiments, physical optical filters are utilized on or withlighting and visualization systems to aid in visualization. For example,physical filters could be applied to the lights to substantially block apredetermined color (e.g., red) from appearing in the imaging.Alternatively, physical filters can be utilized to substantiallyincrease the brightness or coloration of particular color in an image.As such, the physical filters can be applied to block out additionalunwanted properties (e.g., UV, sun photons).

In the exemplary embodiment, system 100 is configured to enable asurgeon and/or user to switch between the filters, combine particularfilters, remove particular filters, and turn the filters on and off alltogether. While the software filters discussed above were provided inthe application of blood during surgery, these techniques could also beused to eliminate other unwanted elements of an image including, but notlimited to, smoke that is released during electrocautery, or movingobjects and debris in the view of the camera. The visualization systemdescribed herein is valuable because the system 100 enables a surgeon tooperate or perform a surgical procedure in a substantially dark roomreducing heat from the lights, which can be detrimental during aprocedure and affect tissue healing. Additionally, the system describedherein eliminates the necessity for water or carbon dioxide air duringan endoscopic procedure.

In some embodiments, imaging device 110 is an ingestible-type camera forexamination of the internal system of a body (e.g. abdomen). Anendoscopic application, colonoscopy, or microsurgery could be used torepair individual nerve fibers or vascular fibers. Aspects of system 100could be used for cardia ablation to localize exactly where irregularcardiac rhythms are coming from.

FIG. 10 is a data flow diagram of an exemplary two-input filteringmethod 700 for use with the system 100 shown in FIG. 1 . In the method700, computing device 102 creates buffered scenes 702 from currentscenes captured by a primary camera (e.g., imaging devices 110). Thecomputing device 102 uses scenes captured by a secondary camera (e.g.,imaging devices 110) to create a background model 704. The secondarycamera may capture images using CT, MRI, infrared, ultrasound, and/orlike techniques. The computing device 102 subtracts the background model704 from the current scene captured by the primary camera utilizing athreshold (T1) to create a foreground mask 706. Moreover, the computingdevice 102 takes the complement of the foreground mask to generate aforeground mask complement 708 (i.e., a background mask). The computingdevice 102 subtracts the foreground mask 706 from the current scenecaptured by the primary camera to generate a background 710. Thecomputing device 102 also generates a foreground 712 by subtracting theforeground mask complement 708 from the current scene captured by theprimary camera. And the computing device 102 subtracts the foregroundmask complement 708 from one or more buffered scenes 702 to generate abuffered background 714. The computing device 102 generates a weightedaverage or threshold (T2) of the foreground 712 and the bufferedbackground 714. The computing device 102 then generates an output frame716 by determining the absolute difference between the background 710and the weighted average.

FIG. 11 is a data flow diagram of an exemplary two-input filteringmethod 800 for use with system 100 shown in FIG. 1 . In the method 800,the computing device 102 creates buffered primary scenes 802 fromcurrent scenes captured by the primary camera and buffered secondaryscenes 804 from current scenes captured by the secondary camera. Thecomputing device 102 uses the buffered primary scenes 802 and bufferedsecondary scenes 804 to find frames where the object in the foregroundobstructing the view is not present. In this embodiment, the computingdevice 102 may create the background model 704 from any combination ofthe primary input, secondary input, output frame 716, primary buffer802, and secondary buffer 804. As described above, the computing device102 utilizes the background model 704 to create a foreground mask 706from the primary camera. In addition to generating the background 710 asdescribed above, the computing device 102 applies the background mask708 to the primary camera input to generate a primary foreground image806. The computing device 102 also applies the background mask 708 tothe secondary camera input to generate a secondary foreground image 812.To create a buffered primary foreground image 808, the computing device102 applies the background mask 708 to images selected from the bufferedprimary scenes 802. And the computing device 102 generates a bufferedsecondary foreground image 810 by applying the background mask 708 toimages selected from the buffered secondary scenes 804. The computingdevice 102 takes a weighted average of the primary foreground image 806over time T2, the buffered primary foreground image over time T3, thebuffered secondary foreground image over time T4, and the secondaryforeground image 812 over time T5. The final output frame 716 isgenerated by the computing device 102 taking the absolute differencebetween the weighted average and the background image 710.

FIG. 12 is a data flow diagram of an exemplary color shift filteringmethod 900 for use with system 100 shown in FIG. 1 . In the method 900,the computing device 102 creates a primary color shift image 902 fromthe primary camera input and a secondary color shift image 904 from thesecondary camera input. In an embodiment, the color shifts are dependentupon one or more wavelengths of interest. The computing device 102applies the background mask 708 to the color shifted primary image 902to create a color shift primary foreground image 908 and applies thebackground mask 708 to the color shifted secondary image 904 to create acolor shift secondary foreground image 910. The computing device takes aweighted average (T2) of the primary foreground image 806, the colorshift primary foreground image 908 threshold (T3), the color shiftsecondary foreground image 910 threshold (T4), and the secondaryforeground threshold (T5). The final output frame 716 is generated bythe computing device 102 taking the absolute difference between theweighted average and the background image 710.

The system and methods described above could be used in tumor, oncology,endoscopic, arthroscopic, and/or tissue ablation procedures such asultrasound ablation. For example, the system and methods could be usedfor guidance, direction, and/or location tasks during surgicalprocedures. In some embodiments the system and methods could be done ona macroscopic level and in some embodiments the system and methods couldbe done on a microscopic level, looking for specific cells, movementpatterns, visualization patterns, and/or electromagnetic patterns. In anembodiment, the system and methods described above could be used todetect cells such as tumor cells or infections, which could be removedwith navigation visualization provided by aspects of the system andmethods. For example, one could have a patient adjust or intravenouslygive a marker that would absorb by abnormal cells such as a tumor orinfectious cells and then visualization aspects of the system andmethods could be utilized to remove pixels or enhance pixels with typesof light frequency, vibrations, and the like. The tumor or infectiouscells could be removed either by external tissue ablation such asultrasonic, thermal guided ablation, or internally. Moreover, it couldguide surgeons during removal of specific cells both on a macroscopicand microscopic level. For example, cells of amyloid deposits forAlzheimer's disease and/or cells that create hypertrophic tissue orinfectious tissue.

While the system and methods described above have been described in anon-limiting medical setting, it should also be noted that the systemsand methods described above (e.g., software filters) could also be usedin non-medical applications, such as optimizing a heads up display on acar when there is fog or rain. In such an embodiment, a vehicle systemcan be configured to utilize the filters described above to filterobjects (e.g., fog, mist, sun, pollution, smoke, or rain) and provide aclear image to vehicle passengers' either in place of a passengers' viewor as an addition to a passengers' view. Such a system is alsoconfigured to detect objects in the path of the vehicle and alertpassengers of the detected objects.

The embodiments described herein enable non line of sight structuresand/or landmarks in the body to be observed before, during, and/or aftera surgical procedure. As compared to at least some known navigationalsystems that require objects to be affixed to the body through invasivemeasures, the systems and methods described herein are capable ofproviding information to a robotic system to enable calibration and/orboundary layer configuration to assist in creating a more efficient andsafe surgical procedure. The methods and systems described hereinprovide surgeons the ability to calibrate a robotic system and performsurgical procedures without direct line of site needed in known systems.The method and systems described herein could vibrate the visual fieldsas particles would have different vibratory frequencies based on theirdensities, thickness, or movement. The particles with variable movementpatterns could then be removed. For example, this could be done throughvibratory, acoustic, electromagnetic, external compression, internalcompression, magnetic frequency, and the like. Although there may be aslight delay, any delay would not affect visualization for surgerytreatment or non-surgical tasks.

The embodiments described herein may utilize executable instructionsembodied in a non-transitory computer readable medium, including,without limitation, a storage device or a memory area of a computingdevice. Such instructions, when executed by one or more processors,cause the processor(s) to perform at least a portion of the methodsdescribed herein. As used herein, a “storage device” is a tangiblearticle, such as a hard drive, a solid state memory device, and/or anoptical disk that is operable to store data.

Although specific features of various embodiments of the invention maybe shown in some drawings and not in others, this is for convenienceonly. In accordance with the principles of the invention, any feature ofa drawing may be referenced and/or claimed in combination with anyfeature of any other drawing. Accordingly, while many proceduresdescribed herein relate to arthroplasty or orthopedic surgery, themethods and systems described herein can be utilized in any surgicalprocedure including, but not limited to, general surgery, cardiothoracicsurgery, tissue ablation, ultrasound ablation, arthroscopic procedures,endoscopic procedures, cardiology and electrophysiology procedures,colon and rectal surgery, plastic surgery, anesthesia, pain managementprocedures, ENT procedures, gastrointestinal surgery, gynecologyprocedures, neurosurgery, oncology procedures, pediatric procedures,radiosurgery, reconstructive surgery, spine surgery, transplant surgery,urology procedures, and vascular surgery. Additionally, it should benoted that the systems and methods described herein could be utilized toprovide encryption technology by determining known patterns and eitheraccepting or rejecting based on a determination that known patterns havebeen detected.

While system 100 has been described as including a procedural component112 and at least one end effector 114, it should be noted that system100 can operate independently to provide visualization and/or navigationto users. For example, system 100 can be utilized in a manual surgicalenvironment where system 100 provides surgical site information to asurgeon operating manually (i.e., without robotic assistance).Additionally, the system 100 described herein can be utilized to providevisualization and/or navigation with other non-medical and/ornon-surgical applications. For example, portions of system 100 andmethod 700 can be installed in vehicles to provide the visualizationand/or navigation needed. Portions of system 100 and method 700 can beutilized to enable a driver/passenger to “see through” objects thatwould limit sight. In the case of a car, truck, motorcycle, bus, orother land vehicle, system 100 and method 700 is utilized to remove fog,cloud cover, rain, sunlight, hail, mist, pollution, smoke, snow, or anyother form of debris obfuscating in air or fluid media from a visualimage to provide a substantially clear image of the path of travel ofthe vehicle. Consequently, the system 100 is configured to provide thesame visualization to air vehicles (e.g., plane, spaceship, rocket,balloon, unmanned aerial vehicle (UAV) (e.g., drone)) and water vehicles(e.g., boats, ships, and submarines). Additionally, portions of system100 and method 700 can be utilized in any application reproducing videoor image feeds including, but not limited to including, residential andcommercial surveillance systems, television production systems andequipment, telescopes, binoculars, marine applications, and satelliteimagery. It should also be noted that the system and method describedherein can be utilized with technologies described in U.S. patentapplication Ser. Nos. 14/451,562, 10/102,413, 13/559,352, and62/275,436, each of which is hereby incorporated by reference in theirentirety.

This written description uses examples to disclose various embodiments,which include the best mode, to enable any person skilled in the art topractice those embodiments, including making and using any devices orsystems and performing any incorporated methods. The patentable scope isdefined by the claims, and may include other examples that occur tothose skilled in the art. Such other examples are intended to be withinthe scope of the claims if they have structural elements that do notdiffer from the literal language of the claims, or if they includeequivalent structural elements with insubstantial differences from theliteral languages of the claims.

A robotic system for navigation of a surgical site is provided. Therobotic system includes a computing device coupled to a presentationinterface, a procedural component, and a communications interface. Thecomputing device is also coupled to a first imaging device configured toprovide imaging data of a surgical site. The computing device is alsocoupled to a second computing device that is configured to provide asecond type of imaging data of the surgical site that is different thatthe imaging data of the first imaging device. The computing device isconfigured to co-register the imaging data to create a surgical siteimage for display to a surgeon on the presentation interface.

What is claimed is:
 1. A system comprising: a robotic mechanism forperforming a navigational operation; an imaging device configured toprovide image data; a computing device in communication with the imagingdevice and the robotic mechanism, the computing device including one ormore processors and at least one memory device configured to storeprocessor-executable instructions, the instructions, when executed bythe one or more processors, configuring the computing device to: receivethe imaging data from the imaging device; differentiate between at leastone object of interest and at least one obstruction in the image databased on movement patterns and color variations; filter the image datato remove the at least one obstruction by altering pixels related to theat least one obstruction; generate an output from the filtered imagedata; and command the robotic mechanism to perform the navigationaloperation based on the generated output.
 2. The system of claim 1,wherein the instructions, when executed by the one or more processors,further configure the computing device to alter the pixels related tothe at least one obstruction based on at least one of (i) one or morebuffered frames and (ii) one or more pixels adjacent to the pixelsrelated to the at least one obstruction.
 3. The system of claim 2,wherein the one or more buffered frames comprise a baseline image andwherein the instructions, when executed by the one or more processors,further configure the computing device to detect the at least oneobstruction based on a comparison of the image data to the baselineimage.
 4. The system of claim 3 wherein the baseline image comprises theat least one object of interest, and wherein the instructions, whenexecuted by the one or more processors, further configure the computingdevice to: mask pixels related to the at least one obstruction; andsuperimpose pixels related to the at least one object of interest overthe pixels related to the at least one obstruction.
 5. The system ofclaim 1, wherein the instructions, when executed by the one or moreprocessors, further configure the computing device to alter the pixelsrelated to the at least one obstruction by recreating the pixels relatedto the at least one obstruction.
 6. The system of claim 1, wherein theinstructions, when executed by the one or more processors, furtherconfigure the computing device to alter the pixels related to the atleast one obstruction by adjusting a color value of one or more of thepixels related to the at least one obstruction based on a color value ofone or more pixels adjacent to the pixels related to the at least oneobstruction.
 7. The system of claim 1, wherein the instructions, whenexecuted by the one or more processors, further configure the computingdevice to alter the pixels related to the at least one obstruction byadjusting an intensity of one or more of the pixels related to the atleast one obstruction based on an intensity of one or more pixelsadjacent to the pixels related to the at least one obstruction.
 8. Thesystem of claim 1, wherein the instructions, when executed by the one ormore processors, further configure the computing device to ceasefiltering the image data or to generate a user alert in response to theat least one obstruction exceeding a predetermined size threshold. 9.The system of claim 1, wherein the robotic mechanism is configured foruse with a vehicle and the navigational operation includes steering thevehicle.
 10. The system of claim 9, wherein the vehicle comprises atleast one of: a car, a truck, a motor cycle, a bus, a plane, aspaceship, a rocket, a balloon, an unmanned aerial vehicle (UAV), aboat, a ship, and a submarine.
 11. The system of claim 1, furthercomprising a display device in communication with the computing device,the display device configured to present the output for visualization bya user.
 12. The system of claim 1, wherein the imaging device comprisesas least one of: a camera, an ultrasound machine, a light detection andranging (Lidar) system, a sound navigation and ranging (Sonar) system,and a laser imaging system.
 13. A method comprising: receiving imagedata; differentiating between at least one object of interest and atleast one obstruction in the image data based on movement patterns andcolor variations; filtering the image data to remove the at least oneobstruction by altering pixels related to the at least one obstruction;generating an output from the filtered image data; and performing anavigational operation of a vehicle based on the generated output. 14.The method of claim 13, wherein filtering the image data to remove theat least one obstruction comprises altering the pixels related to the atleast one obstruction based on at least one of (i) one or more bufferedframes and (ii) one or more pixels adjacent to the pixels related to theat least one obstruction.
 15. The method of claim 14, wherein the one ormore buffered frames comprise a baseline image and whereindifferentiating between the at least one object of interest and the atleast one obstruction in the image data comprises detecting the at leastone by comparing the image data to the baseline image.
 16. The method ofclaim 15, wherein the baseline image comprises the at least one objectof interest, and wherein altering the pixels related to the at least oneobstruction comprises: masking pixels related to the at least oneobstruction; and superimposing pixels related to the at least one objectof interest over the pixels related to the at least one obstruction. 17.The method of claim 13, wherein filtering the image data to remove theat least one obstruction comprises altering the pixels related to the atleast one obstruction by recreating the pixels related to the at leastone obstruction.
 18. The method of claim 13, wherein filtering the imagedata to remove the at least one obstruction comprises altering thepixels related to the at least one obstruction by adjusting a colorvalue of one or more of the pixels related to the at least oneobstruction based on a color value of one or more pixels adjacent to thepixels related to the at least one obstruction.
 19. The method of claim13, wherein filtering the image data to remove the at least oneobstruction comprises altering the pixels related to the at least oneobstruction by adjusting an intensity of one or more of the pixelsrelated to the at least one obstruction based on an intensity of one ormore pixels adjacent to the pixels related to the at least oneobstruction.
 20. The method of claim 13, further comprising ceasing thefiltering of the image data or generating a user alert in response tothe at least one obstruction exceeding a predetermined size threshold.